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(57) ABSTRACT 

An improved visual user interface enables control of the 
interaction of a device with a spatial region. The visual user 
interface includes a display of a control space and a display 
of a target space. The content of the displayed target space 
can be a representation of some or aU of the spatial region. 
The content of the displayed control space and displayed 
target space can also be established such that the displayed 
target space provides context for the displayed control space. 
The device is operated in accordance with the state of the 
displayed control space. The visual user interface enables a 
user to change the display of the control space, thereby 
controlling the operation of the device. Such a visual user 
interface in which a target scene provides context for a 
control scene can, for example, facilitate perception by a 
video camera operator of a current filmed scene in the 
context of the current filmed scene's surroundings, thereby 
enhancing the camera operator's ability to engage in critical 
and sophisticated camera work with great precision. The 
visual user interface can also include a space for maldng 
annotations regarding the interaction of the device with the 
spatial region, thereby enabling such annotations to easily be 
made contemporaneously with control of the device, so that 
a single person can perform both functions well. 
Additionally, the annotation capabiUty can be implemented 
in a way that enables the user to handwrite the annotations, 
thus facilitating a wide range of expression in creating 
annotations and avoiding the problems associated with voice 
annotation systems. 
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VISUAL USER INTERFACE FOR USE IN camera operation allows the camera operator to move away 

CONTROLLING THE INTERACTION OF A from the vicwfinder so that the entire area sunounding a 

DEVICE WITH A SPATIAL REGION current filmed scene can be more easily o bserv ed during 

filming. However, the(caniera7operatb£^tiircannot,sima^^ 
5 neously^^iew: the sunoundiiigTafealand^^^ filmdS 
BACKGROUND OF THE INVENTION soene,tso that such remote camera operation still suffers 

from the above-described deficiencies to an undesirable 
degree. 

, ... r J • -.1. .1 • The above-described operation of a video camera is 

controlling the mteraction of a device with a spatial region, ,„ 4 - A. * * 

. 5* . ^ 1 . .J J ■ JL. 10 manual operation, i.e., the camera operator must either 

such as the ^acquisition or display by a video device o>a , , _ , . 

r J - /TY ^ . — r - — - jc? directly or mdirectly provide control of the camera at each 



1. Field of the Invention 

This invention relates to a visual user interface for use in 



sequence of filmed scenes that are;cach part of a panbramiS^^ 
f scene,^ 



instant in time. Automatic control of a video camera has also 
previously been implemented. At a predetermined location, 
2. Related Art ^ video camera is moimted on a camera support that allows 

A conventional video camera typically includes a ^5 a variety of camera motions. The video camera is connected 
viewfinder, the view through which corresponds generally to to a control device as described above. Metrics are estab- 
a portion (the fihned scene) of a scene that is being fihncd. Ushed for the camera support so that positions of the camera 
During conventional operation of such a video camera, the support can be correlated with particular filming directions 
camera operator looks through the viewfinder while filming of the video camera. During operation of the video camera, 
so that camera operator knows what is being filmed and so 20 the control device, in accordance with a pre-specified 
that the camera operator can control the camera to film a instructions, transmits control signals to the motors to cause 
desired sequence of fihned scenes. However, during such the camera support to be moved in a fashion that produces 
operation, the camera operator's view is limited to what can a desired sequence of filmed scenes. Using such automatic 
be seen through the viewfinder. Since the camera operator control, the position, velocity and acceleration of the video 
cannot view portions of a scene adjacent to the filmed scene, ^ camera can be controlled at each instant in time. Automatic 
the camera operator is deprived of context for the filmed control of a video camera can enable dynamic control of the 
scene. This limitation impairs the camera operator's ability camera that may not be reproducible by a human operator 
to operate the camera so as to film a scene in a desired way with the same precision or speed. Automatic control can also 
(i.e. to acquire a desired sequence of filmed scenes). It has enable pre-scripting of a sequence of filmed scenes, 
historically been part of the art of a skilled camera operator 3Q However, automatic control still does not satisfactorily 
to be able to remember the content of a scene having a scope address the above-described problems, since it does not 
larger than can be encompassed within the field of view of allow real-time camera control in a manner that facilitates 
the viewfinder of the video camera, so that the camera perception by the camera operator of the filmed scene in the 
operator can seamlessly control the camera to change the context of the fihned scene's surroundings. Filming a live 
fihned scene in a desired manner. 35 event, such as a sporting event, is just one example of a 

A video camera can he hand- held. A video camera can situation in which such automatic camera control is inap- 
also be mounted on a camera support device that typically propriate. 

provides multiple degrees of freedom of movement of the Once a videotape (or other set of video data) has been 

camera. The above limitation in operation of a conventional produced by filming a sequence of video scenes, it is 
video camera applies to both hand-held and mounted camera 40 desirable to have an indication of the content of the video- 
operation in which the camera operator controls the content tape and, further, an indication of where (i.e., when) par- 
of the filmed scene by looking through the viewfinder. While ticular content occurs within the videotape. A convenient 
the camera operator can periodically look away from the way to create a record of the content of a videotape is to 
viewfinder to observe the larger context of the scene being produce a set of timefidentified annotations7dunng filmid| 
fihned, such operation is undesirable for a variety of reasons. 45-pfithe^vide6t%>e. However, many videotaping systems do 
For example, the camera operator may inadvertently misdi- not include a capability to produce such annotations. Cre- 
rect the camera while not looking through the viewfinder. It ation of a handwritten set of annotations by the camera 
is ako difficult for the camera operator to readjust image operator using, for example, paper and pencil can unduly 
perception and focus each time the operator looks away interfere with the actual filming, thus necessitating the 
from the viewfinder to the larger scene, and vice versa. 50 presence of a second person to make the annotations. Some 
Additionally, the inabiUty to view a current filmed scene videotaping systems include a user interface that enables 
simultaneously with the surrounding context of the current selection of one of a set of predetermined textual annota- 
filmed scene makes it difficult for the camera operator to tions. However, such systems can be inadequate because 
plan movement of the camera to fihn other scenes adjacent they necessarily enable only a Umited range of description of 
to the current filmed scene. 55 the content of the videotape. Further, these systems, while 

Mounted cameras can also be operated remotely. A video typically more convenient than using paper and pencil, may 
display at a remote site is connected to the video camera so also require the presence of a second person to perform the 
that a cu5rCTt:filmed^ceneis di^layed by^^^ annotation so that the camera operator is not inordinately 

(^display. A control device is connected to, for example, one distracted from filming. Some videotaping systems enable 
or more motors (or other driving devices) which are, in turn, 60 voice annotation to be made during filming of a videotape, 
coupled to the camera supportCso as to move ffie camSFa However, voice annotation is inappropriate for situations in 
(Support-in^esponse-to-sipaJs received from t^^^^ ^nti^ which speaking during filming of the videotape may disrupt 
deviflt. The camera operator observes the remote video the event or scene being filmed. Further, searching a set of 
display and operates the control device to control the audio data (here, the set of spoken annotations) for desired 
sequence of filmed scenes acquired by the video camera. 65 content can be difficult, therefore making it difficult to 
Aside from the obvious benefits (e.g., the capabiUty of review a set of annotations to find a particular description of 
hlming in areas that are hazardous to humans), remote video the videotape content. 
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SUMMARY OF THE INVENTION The invention can be used in a wide variety of applica- 

In accordance with one aspect of the invention, control of ^ions. In particular, the invention is useful in applications 

the interaction of a device with a spatial region is enabled which require the control of a video device. Examples of 

using an improved visual user interface. The visual user such apphcations with which the invention can be used 

interface includes a display of a control space and a display 5 include movie and television film production and display, 
of a target space. cThi^target*spSbei;typic2diySs_ a speeifled^ security camera systems and ^idTOroonfercncing'systen^ 

rclMionsHip toZtherspatialjegionraii^ The invention can be used, however, in applications other 

tation of some or all of tfae^spatialTegjcffr. The content of the than those in which a video device is controlled. For 

control space and target space can also be (and often is) example, the invention could be used in controlling a 
established such that thectMget spa^ provides:.cont^ fcff^" lo lighting device or devices so that a spatial region is illumi- 

the control spac^ The device is operated (i.e., interacts with uated in a desired way. 
the spatial region) in accordance with the slate of the display 

of the control space. For example, the device can be operated BRIEF DESCRIPTION OF THE DRAWINGS 

in accordance with the spatial relationship between the ^ . . ..^ j , ^ . ... 

control space and the target space. The visual user interface ^IG. 1 is a simplified rcprescntaUon of a system with 

according to this aspect of the invention enables a user to accordance with the mvenUon. 

change the state of the display of the control space, thereby FIG. 2 illustrates a visual user interface, m accordance 

controlling the operation of the device. Generally, the dis- with one embodiment of an aspect of the invention in which 

play of the control space can be changed in any manner that a visual user interface is used to control the interaction of a 

is possible for the application with which the visual user ^ device with a spatial region, that is used to control the 

interface is being used. operation of a video device. 

For example, a Yisualtuiei^inSSace- accordin^^ FIG. 3 is a panoramic scene within which a sequence of 

Cai^ed^^f^ttielinveMon^an-di^lay7a-targ^ scenes is to be fihned or displayed, 

controrsceoe. Such a visual user interface can be used, for FIGS. 4A through 4C illustrate successive filmed scenes 

example, to control a video device. The video device con- 25 from the panoramic scene of FIG. 3. 

troUed can be of any type, including video devices that FIGS. 5A through 5C each Hlustrate a visual user interface 

acquire video data (e.g., video cameras, videonenabled accordance with the invention at the time that the corre- 

motion picture cameras) and video devices that display spondingoneof the scenes ofFIGS. 4A through 4C is being 

video data (e.g. , televisions, computer display monitors, and filmed 

video or graphi^ projection systems). A .uscr_cai^ for 30 ^^^^ 6A through 6C iUustrate the operation of the visual 

/ example, cause the c ontro -scene-to;chanf; e:_m .size^^^ ^ accordance with an embodiment 

■ angular onentation and/or location with respect to the target . . u- u *u ^ • * ^ 

^ c — 1 - -J - - of the mvention m which the area of view of the target scene 

sQEnc,_whidi can cause, for cx^niple«.a filmed scene tcP , , ^ , *i « 

\^ ' . . — J -r^^ ■ TU^ cau move as necessary to keep the control scene at least 

changc'^in 'a'^eorrcspondiiigtwS}^® ther^asp^ts t)f the contror ^ au 1 4 ♦ 

~ L I' 7^ partly surrounded by the target scene, 

scene; sucb_ as visual -parameters^ (e.g.,~focus 35 j t> „ , , 

bcightnc^),S:"an alsb'be changS. '^^ ^^^^g^ illustrate the use of a stylus and 

A'visual user interface according to this aspect of the touchscreen to rnalce spatial changes in a ajntrol scene that 

invention can overcome, for example, the above-identified ^^^^^ corresponding spaUal changes m a filmed scene, 

deficiencies in previous video camera operation that are FIGS. 8A and 8B illustrate a visual user interface in 

associated with the limited field of view of a video camera 40 accordance with an embodiment of the invention in which 

or instance, a visual user interface according to the invention current fihned scene is displayed in addition to the 

in which the target scene provides context for the control control scene and the target scene. 

scene can fadUtate perception by a video camera operator of FIG. 9 illustrates a visual user interface in accordance 

a current filmed scene in the context of the current filmed with another aspect of the invention in \Wuch a visual user 

scene's surroundings. A camera operator's ability to engage 45 interface includes a space in which a user can make anno- 

in critical and sophisticated camera work with great preci- tations regarding the device interaction with the spatial 

sion is gready facilitated. The visual user interface according region. 

to this aspect of the invention can enable such difficult tr r?t>. r>t-oi-.T>Tm-T/-kKT /-utttjc 

camera work to be performed by relatively unskilled camera DETAILED DESCRIPTION OF THE 

operators who heretofore would not have been able to 50 UNVliINJlUM 

perform such camera work acceptably. Stated generally, a visual user interface according to one 

In accordance with another aspect of the invention, a aspect of the invention can enable control of the interaction 

visual user interface embodied in a single device enables of a device with a spatial region. (Herein, such "interaction" 

control of the interaction of a device with a spatial region can also include the display of an interaction of a device with 

and the capability to make annotations regarding the inter- 55 a spatial region.) The visual user interface ^ioclird^^^pt^ 

action of the device with the spatial region. The visual user CofxcontroTsp^ce andja target.^ 

interface according to this aspect of the invention enables ispace:hasxspecifi^Telationship to^^d^^ 

such annotations to easily be made contemporaneously with th? Jarget gia^-can represent :some or aH of the'^patSIl 

control of the device, so that a single person can perform refiicpCTbg control^ace and target space can^be displayed 

both functions weU. Addidonally, the visual user interface 60 together. _sp ttiat at least a portion of the control spite 

can be implemented in a way (e.g., with a touchscreen and ove rlaps^ at least^ a portion of tJbe target "^ac^theTrontent 

stylus) that enables the user to handwrite the annotations, tfieloveilapping^portioDsro^^^^ 

thus faciUtating a wide range of expression in creating sp^re;bcing substantiaUy the s^ 

armotations. Such a visual user interface also avoids the pr ovides' co ntext:for, the overlapping portionTof tKc^coritiol 

problems associated with voice annotation systems, e.g., 65 Upac^ The device is controlled in response to the state of the 

disruption of the event or scene being filmed, and difficulty display of the control space. For example, the spadal rcla- 

in searching through the annotations. donship between the control space and the target space can 
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be determined and used to control the device in a particular the user interface device 104 in a portable device is advan- 

tnanner. The visual user interface enables a user to change tageoiis becaxisc it enables the user interface device 104 to 

the state of the display of the control space, thereby cflfecting be easily moved to filming locations together with the video 

control of the device and its interaction with the spatial camera 101 and tripod 102. As illiistrated in HG. 1, the tiser 

region. 5 interface device 104 includes a portable computing device 

A visual user interface in accordance with this aspect of 105, as well as a stylus 106 that can be used to contact a 

r::the:inven&^ Hn be used, for exam touchscreen (which can be used to display the visual user 

^deyi<^. Herein, "video device" encompasses any device for interface according to the invention) of the portable com- 

acq\iiring video data and/or generating a display from video puting device 105 to effect input of instructions from the 

data (e.g., a video camera, a video-enabled motion picture user to the portable computing device 105. However, the 

camera, a television, a computer display monitor, and a user interface 104 can be implemented by other devices, 

video or graphics projection system). Hereafter, often, the including computing devices that arc not portable, such as, 

invention is described in detail with respect to embodiments for example, a desktop or workstation computer, 

of the invention that can be used with video devices. It is A data processing and storage device 103 is intercon- 

understood, however, that the invention has broader appli- 15 nected with the user interface device 104, the tripod 102 (in 

cability and can be used with other types of devices, some particular, a position encoder or encoders of the movable 

of which are also discussed below. camera mount) and the video camera 101 to enable com- 

FIG. 1 is a simplified representation of a system 100 with munication therebetween. Such communication can be 
which the invention can be used. A video camera lOl-is^ implemented using any appropriate methods and apparatus 

mounted on a-moyable camera-moimt q 20 (^ g - serial or parallel digital communications apparatus and 

other^ame^ support device):^ protocols, such as the RS232/C protocol and associated 
can:b.e:embodiea:hy-any.^ch: mount (of; apparatus), as known to those skilled in the art. Moreover, 

knownr:types :exist) that enable s desired- mo vemei^j such communication can be either wired or wireless, thus 

(f6lational,~translational or-bolh):of te^ enabling the data processing and storage device 103 and the 

Often, movable camera mounts are implemented to enable 25 ^^er interface device 104 to be implemented either as 

one or more of three types of rotational movement of a video tethered (wired) or untethered (wireless) devices. A system 

camera; panning (i.e., rotation about a panning axis that iu which the data processing and storage device 103 and the 

changes the filming direction of a video camera in a right or user interface device 104 are untethered can be advanta- 

left direction relative to a "normal" orientation of the video geous because it affords greater freedom of movement for a 

camera), tilting (i.e., rotation about a tilt axis that changes 30 during use of those devices (and, in particular, during 

the filming direction of a video camera in an up or down use of a visual user interface according to the invention), 

direction relative to a "normal" orientation of the video The data processing and storage device 103 can, for 

camera) and rotating (i.e., rotation about a rotational axis example, store video data acqudred by the video camera 101 

that leaves the filming direction of a video camera and data describing the video camera metrics discussed 

imchanged or substantially unchanged and, instead, causes 35 above, as well as any other data and/or instructions (e.g., 

rotation of a filmed scene). Advantageously, the movable audio data and instructions for use of audio data) that are 

camera mount can enable movement of the video camera used in the operation of the system 100. The device 103 can 

101 having three rotational degrees of freedom (i.e., rotation also process instructions and data received from the video 

about the pan, tilt and rotational axes), and the invention is camera 101, the position encoder or encoders of the movable 

often described below with respect to embodiments that 40 camera mount of the tripod 102 and user interface device 

enable such movement. However, the invention can also be 104, as well as transmit instructions^and data to those 

used with systems in which the video camera movement has devices, fEoT^xampleT-wHen^STinstnictionTrega^ing^t^ 

only one or two rotational degrees of freedom. The invention ^filjfflLng :directio nrpositiori and/or oTientationof-jhe-vide^ 

can also be used with systems that enable translational camerarl01:is:received:frDm:the:ti^ 

motion (one, two or three degrees of freedom) of the video 45 the:device"103:uscs:the:mctric"inforii2ati6ir stored therein t6 

camera in addition to, or instead of, rotational movement. determine^tKe^^propfiate3pbsitiori~6f the-mov 

(For example, a particularly robust system according to the iiiount3and:send an'instniction.to : cause the movable :cairiera 

invention could be implemented to allow video camera mouiit-tb-move to= that positions Or, for example, the device 

motion having three rotational degrees of freedom and three 103 can receive video data representing filmed scenes 

translational degrees of freedom.) As explained in more 50 acquired by the video camera 101, process and/or store the 

detail below, metrics are established for the movablecamera video data as necessary or desirable, and communicate data 

mount that ^rorTelatcTpositio5sro&Uae*movaM and instructions to the user interface device 104 to cause 

with"filiriirig:dij^lions7b'ri^titie^*an^^ generation of a display of a visual user interface according 

cyidcoxarnera lOl*. (A filming direction can be specified in to the invention, as described elsewhere herein. The device 

any appropriate manner, such as, for example, by specifying 55 103 can also store data representing the camera metrics for 

the content at the location of the center of the area being each filmed scene acquired by the video camera 101. 

fihned by the video camera.) The device 103 can be implemented, for example, by a 

A user interface device 104, which can be any appropriate conventional digital computer. In particular, the device 103 

device that can display a visual user interface according to can advantageously be embodied in a portable computing 

the invention, is used by a user to input instructions for use 60 device (examples of which are given above with respect to 

in controlling operation of the video camera 101. The user the description of the user interface device 104). As with the 

interface device 104 can be embodied by, for example, a user interface device 104, embodying the data processing 

conventional portable computing device, such as a notebook and storage device 103 in a portable device is advantageous 

computer, subnotebook computer, personal digital assistant because it enables the data processing and storage device 

(PDA) or other similar device, together with, as appropriate, 65 103 to be easily moved with the video camera 101 and tripod 

oniriorl&re^asMciated uscr.^^ such as, for 102 to filming locations. However, like the user interface 

example, a mouse, keyboard, trackball or stylus. Embodying device 104, the data processing and storage device 103 can 
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be embodied by devices that are not portable, such as, for ^ten-thc coDlroUed-device-is-a'^vidco-de^^ 

example, a desktop or workstatioa computer. contcnt^^a^Jtoed^s^nC:^^ 

The data processing and storage device 103 and the user entirely) drawn from the target scene. Often, a targct'sc^ r 

interface device 104 can advantageously be embodied in the ^represents a "panoramic scene,'' which, as used her^^( 
same device, thus reducing the number of separate pieces of 5 f I'efers to a scene that can entirely encompass a filmed scene^j 
apparatus necessary to implement the system 100. ^ practice, a panoramic scene may be much larger than 

FIG. 2 illustrates a visual user interface 200, in accor- pmed scene. For example, the target scene may represent 

dance with one embodiment of this aspect of the invention, panoramic scraie which encompasses the entire field of vie^ ^ 

that is used to control the operation of a video device, such of a camera operator, or a panoramic scene which cncom^ 

as the video camera 101 of FIG. 1. (The visual user interface 10 P^^^ ^ ^^^^ ^^^^ ^ specified location 

200 can also be used in controlling other types of devices, the target scene represents a panoramic scene, il 

as discussed further elsewhere herein.) The visual user target scene can be a "composite scene," which is a scei 

interface 200 includes a displaj^:'of:a: targ'et:s cene _201-and-a> fhat is constructed from a combination of multiple filmec^ 

rd ispiay-of-a-control sceneloZ As^^^ scenes. Below, particular aspects of the invention ar^ 

^IroEscene'SiFl^^ 15 described with respect to an implementation of the inventio|''^ 

^IrJherconteHTonhe^ in which the target scene is a composite scene that represent 

substantially-the^^^the:co_ment of:the_target:s^^ a panoramic scene, and the control scene is the same (oj^ 

^t@lOTffl'cftheiwise.be^ rQearly the same) as a c^ent filmed scen e (see, for example; 

soene^Ol^^rsuc^^^ [FIGS. 5A through^SB)^ ' 

rprgyid^^^g^dlfdCth^^ 20 ^ a panoramic scene within which a sequence of 

^better-appredated in-the-iUustration of^aAosi^^ scenes ("filmed scenes,"' in accordance with the above 

arcogmg!tocth^iMiMon.sh^^ definitions) can be filmed or displayed. To illustrate the 

^described-belowv)-The-boundaries^ advantages of the visual user interface according to the 

,ca5^^elineated"in^yiappro"pn aspect of the invention now being described, it will be 

^'known^y-those skiUed-in-therartrofiimpleme^^ 25 assumed that a camera operator wishes to film the panoramic 

^dj^J^p- scene of FIG. 3 by beginning filming near the upper left 

CA^filmedlcene is acquked or displayed by the vide7f co™er of the panoramic scene, then sweeping downward to 

device in accordance with the state of the display of ther the center of the panoramic scene, and finishing by sweeping 

control scene 202. As described in more detail below, the^ upward to the upper right comer of the panoramic scene. It 

^ser can change the display of the control scene 202 to cause 15o ^ be understood that the principles of the invention 

changes in the filmed scene. For example, if the visual user k illustrated in the discussion below are also applicable to use 
interface 200 is implemented in the user interface device 10^ * ^^er interface according to the invention to 

(FIG. 1), changing the position of the conU-ol scene 20^ display a sequence of fihned scenes generated from a set of 

■within ihfe target scene 201 can cause an instruction to b video data that represents the panoramic scene of FIG. 3. 
^sent to the position encoder or encoders of the movableWss FIGS. 4A through 4C illustrate successive filmed scenes 

camera mount of the tripod 102 to cause an appropriatJl. fi-om the panoramic scene of FIG. 3 obtained during fihning 

change in^the fihning direction and/or position of the videos as described above. In previous manual control of a video 



camera ld|.^ — ^ - — camera, as described above, at any particular time the 

'^HereinTa "fihned scene" is a scene that is being, has been, camera operator can look at either the filmed scene or the 

or can be acquired or displayed by a video device. As used 40 panoramic scene, but not at both simultaneously. Such video 

herein, filmed scene typically refers to a representation (e.g., camera control is disadvantageous, as described in more 

a frame of video data) of a part of a spatial region (which can detail above. For example, if the camera operator chooses to 

be the entire spatial region), though the term can also be used lool^ at the fihned scene while fihning, then, while the 

to refer to the part of the spatial region itself. A "control camera operator is fihning the scene of FIG. 4A, for 

scene" is a displayed scene that is displayed as part of a 45 example, the camera operator sees only that scene. The 

visual user interface according to the invention and can be camera operator must rely on his memory of the entire 

changed by a user to effect control of a device, (Similarly, a panoramic scene of FIG. 3 to know how to move the camera 

"control space" is a displayed space that is displayed as part to get to the scene of FIG. 4B in a desired way (e.g., with a 

of a visual user interface according to the invention and can particular sweeping arc). Moreover, the camera operator 

be changed by a user to effect control of a device.) When the 50 must rely on such memory to recall the content of the scene 

device is a video device, changing the conu-ol scene causes of PIG. 4B itself, i.e., to know where he wants the camera 

a change in the fihned scene. In such an embodiment, often, to go. As can be appreciated, this can be a daunting task, 

the content of the control scene is the same (or substantially particularly if the camera operator must fihn for an extended 

the same) as the content of a current fihned scene, though, Period of time without a break (i.e., without looking away 

as described below,_this.neednot.necessaril y be the case^ 55 frotn the filmed scene). 

r"iai^erscei[i?*lsT display ed-scene-thatis-displayed-as-part FIGS. 5 A through 5C each illustrate a visual user interface 

^of-a-^visuahuse^^ 500 according to the invention at the time that the corre- 

provides-contexTfor a-control scene-and/orTirfela^ spending one of the filmed scenes of FIGS. 4A through 4C 

i^ntrol:s<xnclt6::effectlpaxtic^ is being fihned. Like the visual user interface 200 of FIG. 2, 

ibeingSatroll^. (Similarly, a "taiget spa^^ is a displayed 60 the visual user interfaceC5jO|Pimclud es'a-disp _lay/of ajpntro 

space that is displayed as part of a visual user interface fsccn<r502"tliatris^6fii3edregion^ 

according to the invention and provides context for a control scene _ SOl^IsuclCthat^^the^OTntent^p 

space and/or is related to a control space to effect particular provide s con tcx t for the con troLsc6n'6l5Q2l 

operation of the device being controlled.) The target scene As shown in FIGS. 5A through 5C, in the visual interface 

typically has a specified relationship to the spatial region 65 500rthc ;control"sce ne 502 is' the" same js'the filmwi^^Be 
with which the controlled device interacts^^andjiq^en^'is'^ .(cxampl^of wMchlSe^own in H 

representatioDrofrsornc-or-^alirof-tl^^ ^le^yytgeO^eneTSOlTisIthe-same as-the-p^^ 
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(the control sceneI502 . For example, as illustrated in FIGS. 
5A through 5C, the control scene 502 (and, thus, the filmed 
sceoe) can be moved to various locatioiis within the target 
scene 501. In accordance with the aspect of the invention 
now being described, then, during filming of the panoramic 
scene of FIG. 3 as described above, the camera operator 
would look at the visual user interface 500 illustrated in 
FIGS. 5A through 5C, rather than through the viewfinder of 
the camera (direct manual camera control) or at a display of 
the filmed scene (remote manual camera control), as is 
depicted in FIGS. 4A through 4C. The benefits of the visual 
u ser interface 500 are readily apparent. jg~gie earner; 
operator moves from filmed scene to fildt 

(dis^^^^^^^^^roi ;i^^|^5 O^^igjalv^^ 
.kne*o,u^y^^^v^d^ 

"oramicsceW as Ih^ J[aJg^ s^eD<i_56l) 

^In~additioh"to"being"iiscd to control operation of a video 
camera, the visual user interface 500 could also be used to 
control the display of an existing set of video data that 
represents a panoramic scene. Again, the user would make 
changes in the control scene 502, such changes causing 
corresponding changes in the displayed filmed scene. 

While, as indicated above and by the detailed description 
of the invention below, the invention is particularly useful as 
applied to control a video device, the invention is not limited 
to such applications. In general, a visual user interface 
according to the invention can be used to control the 
interaction of any type of device with a spatial region and 
can control the device in any manner or t o perform^ y 
function, such as, for example, 

^iSS^^^ba^cmgJ 
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"For example, a visual user interface according to the 
invention, such as the visual user interfaces 200 (FIG. 2) and 
500 (FIGS. 5A through SC), could also be used to control a 
lighting device or devices so that a spatial region is illumi- 40 
nated in a desired way. A user could, for example, position 
a control scene within a target scene to cause a spotlight to 
be directed to a particular location within a spatial region 
represented by the target scene. Making the control sceoe 
larger or smaller could, for instance, cause the emitted light 45 
to become focused on a correspondingly larger or smaller 
area. Changing the focus of the control scene might, for 
example, increase or decrease the colUmation of the emitted 
light, while changing the brightness of the control scene 
could cause corresponding changes in the intensity of the 
emitted light. The above-described techniques for using a 
visual user interface according to the invention to control a 
lighting device arc merely illustrative; they are neither 
necessary nor exhaustive. 

In general, a control space displayed as part of a visual 
user interface according to the invention can be changed in 
any manner. For example, a user can cause a display of a 
control space (e.g., a control scene) to change spatially with 
respect to a display of a target space (e.g., a target scene), 
such change causing a predetermined change in the opera- 
tion of a device (e.g., a filmed scene of a video device). As 
one illustration of such spatial changes, a user can cause the 
display of a control scene to change in size, shape, angular 
orientation and/or location with respect to the display of a 
target scene. As used to control operation of a video camera 
(or other video device), for example, (Sji^LcKangingrof^tlft 
cont rol'scene"c aD:b'£^manifested^,rfor exainplerpaimi^ 
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tiltingj-rotatmg^md/orz^omiiig. A user can also change the 
di^liyof a control space in other ways. For example, a user 
can change visual parameters (such as focus, tint, contrast 
and/or brightness) of the display of a control scene. As used 
to control operation of a video camera or video enabled 
motion picture camera, for example, such changing of the 
control scene can be manifested as, for example, changes in 
the lens focusing mechanism, white balance, pedestal, and/ 
or aperture and/or gain, respectively, of the camera. 

A control space (e.g., control scene) can, in general, have 
any predetermined relationship to the device interaction with 
the spatial region (e.g., filmed scene). For example, in a 
visual user interface according to the invention that is used 
to control a video device, it is anticipated that it will often 
be advantageous to have a control scene that is the same (or 
substantially the same) as the filmed scene. However, this 
need not necessarily be the case. The control scene can be 
larger or smaller than the filmed scene. For example, it may 
be desirable to make the control scene proportionally larger 
or smaller than the filmed scene. 

Further, in general, the interaction of the device with the 
spatial region (e.g., filmed scene) can change in response to 
changes in the display of the control space (e.g., control 
scene) in any predetermined manner. For example, in a 
visual user interface according to the invention that is used 
to control a video device, it is anticipated that it will often 
be advantageous for the filmed scene to change in the same 
maimer as the control scene, e.g., if the control scene is 
changed in size, the filmed scene is changed in size in the 
same proportions (and, perhaps, by the same amount or 
amounts). However, again, this need not necessarily be the 
case. 

As indicated above, in a visual user interface according to 
the invention that is used to control a video device, the 
filmed scene can advantageously be the same or substan- 
tially the same as the control scene (as in the visual user 
interface 500 described above), and can advantageously 
change in a manner that directly corresponds to changes in 
the control sceoe. Such a visual user interface is 
advantageous, for example, because it provides the user with 
a visceral sense of the control being effected. 

Additionally, the target space (e.g., target scene) can, in 
general, have any predetermined relationship to the control 
space (e.g., control scene l'^Fof e xarcLplcZin^ajyisuahuscr^ 
interfoce-arcqrtog_to thCinyeMoK^ 
yidedZd^ceT^itTisTantiapg ^ 
^^geouslforlthcipontfolIs^De-to-be-encompa^ 
Centirely,~^utrat-Jleast:^^^ This is 

advantageous because it provides context to the user that can 
aid the user in changing the control scene in a manner that 
provides desired control of the filmed scene. Again, though, 
there need not necessarily be such a relationship between the 
control scene and target scene. 

In general, bothi theidis plaV of 3he con trol spaoe'and'tfa^ 
^iMayijQfit^target'ai^^ or a^ 



fdyflSmieSlilgSy.^ It is anticipated that, in many applications 
of a visual user interface according to the invention, it will 
be advantageous to have a dynamic control space display. 
When a visual user interface according to the invention is 
used to control a video device, for example, this can usually 
be easily accomplished by making the control scene the 
same as the filmed scene, which, typically, wiU be djmamic. 
Additionally, in a visual user interface according to the 
invention that is used to control a video device, whether a 
static or dynamic target scene is used will typically involve 
a tradeoff between the cost of generating the display of the 
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target scene (static is typically cheaper) and the quality of j-^^^mation^^ofjm^^ To form the com- 

thc display of the target scene (dynamic generally provides posite sceneTMJanoramic scene that is to constitute the 

a more accurate display). target scene can be scanned by a video camera that records 

Above, a visual user interface, according to one aspect of filmed scenes (i.e., video "frames") at the frame rate of the 

the invention, that enables control of the interaction of a 5 video camera (e.g., 30 frames per second). (The filmed 

device with a spatial region has been described generally. scenes could also be acquired by scanning the panoramic 

Below, a detailed description is given of ways in which a scene or defined parts of the panoramic s cene with m ultiple 

visual user interface according to this aspect of the invention video cameras.) [Thej51ii^^~g ^nesland/or-parts^o f,filin^ 

can be implemented. The description is generally made with sqen cs~are "^ en^Icombined-in" some-^m anner- to -produce- tnb 

respect to a visual user interface that is similar to the visual jq COTnpoateIg:engjnbeIfil^^ 

user interfaces 200 (FIG. 2) and 500 (FI GS . 5 A through 5C) , [anatogidata-and^thc^meth^ 

i.e., a visual user interface including i) a target scene that opwatesjDnldig^talTdata^tlius^ 

represents a panoramic scene, and ii) a control scene that is fconv;6rsiQn,ortlierdata^rhis can be accomplished, as known 

srnaller than, and subsumed within, the target scene, and that by those skilled in the art, using conventional methods and 

is, when the visual user interface is used to control a video ^5 apparatus. The video camera or cameras used to acquire the 

device, the same (or nearly the same) as a fihned scene. fihned scenes from which the composite scene is to be 

To enable the spatial characteristics (e.g., location within created can be panned, tilted, rotated and zoomed to enable 

the target scene, and angular orientation, size and shape the fiill resolution of the camera or cameras to be used in a 

relative to the target scene) of the control scene to be used variable and cumulative manner, thereby enabling the com- 

to control operation of a video device, metrics must be 20 posite scene to be constmcted from multiple filmed scenes 

recorded that relate particular content in the target scene to which each have up to the full resolution of the camera or 

particular physical attributes of the device being controlled. cameras. 

For example, if the device being controlled is a video For example, the fihned scenes can be "stitched" together 

camera, then, for a particular camera location relative to the to form the composite scene. Each filmed scene represents 

panoramic scene, the location of a region of content within 25 a portion of the panoramic scene. The content of the filmed 

the target scene (Le., the filming direction) can be related to scenes can, and typically wiU, overlap. Stitching the filmed 

a mechanical position of the video camera (e.g., the angular scenes together causes the content of the filmed scenes to be 

orientation about the pan and tilt axes of a movable camera pieced together so that corresponding overlapping parts of 

mount of a tripod-mounted video camera). Additionally, the filmed scenes are blended together to make the composite 

angular orientation of a region of content can be related to 30 scene appear to be (relatively) seamless. Stitching together 

a mechanical position of the video camera (e.g., the angular of multiple filmed scenes to form a composite scene can be 

orientation about the rotational axis of a movable camera accomplished using any appropriate method. Since methods 

mount of a tripod-mounted video camera). Further, the size that produce the highest quality composite scenes are usu- 

of a region of content can be related to the camera lens ally the most computationally expensive, the particular 

characteristics (e.g., the setting of a zoom lens). It may also 35 method used may be chosen based on a tradeoff between 

be possible to relate the shape of a region of content to computational cost and quality of the composite scene, 

camera characteristics: for example r t hc~as pcct~ratio~of23f Frame stitching methods that can be used with the invention 

rectangidar^gion"ofIc6atCT[trcaji:bt"rclatedIt o~^ are described in, for example, U.S. Pat. Nos. 5,396,583, 

ra§o3pf;therima^ginglde,vi^Iofltl^^ 5,510,830 and 5,262,856, the disclosures of which are 

priatd y subsamplc t hc:filnied^ene.reoord6d-by[tKc^ 40 incorporated by reference herein. 

The determination of such video camera metrics can be A composite scene can be formed from multiple filmed 

accomplished using any appropriate method. For example, scenes in any other appropriate manner, as well. For 

one such method is described in U.S. Pat. No. 4,847,543, the example, the video camera metrics can be used to locate 

disclosure of which is incorporated by reference herein. As each filmed scene within the panoramic scene. Filmed 

indicated above (see the description of FIG. 1), the video 45 scenes can then be selected, based upon their location, and 

camera metrics (i.e., encoded positions of the video camera) pieced together like a puzzle so that each selected filmed 

can be stored in a data storage device and used to control a scene adjoins or overlaps other adjacent selected filmed 

position encoder or encoders of a movable camera mount to scenes. However, the matching at the edges of the filmed 

appropriately position the video camera in response to scenes may not be exact, so that a composite scene formed 

spatial changes in a control scene made by a user of a visual 50 in this way may not appear as seamless as a composite scene 

user interface according to the invention. formed by stitching. 

If, instead, the video device being controlled is a video For many applications with which the invention can be 

display device, then the location, angular orientation, size used, it is preferable that the filmed scenes from which a 

and shape of a region of content (i.e., the content of the composite target scene is constmcted be acquired prior to 

control scene) within the target scene can be used to sub- ss use of the visual user interface to control a device, since, as 

sample video data that is used to generate the display of the indicated elsewhere herein, the target scene is often used to 

target scene. In many cases, the video data used to generate provide context for the control scene. This can be done, for 

the target scene wiU have previously been acquired and example, by using a video camera in a conventional maimer 

stored on an appropriate data storage device. However, it is to film a panoramic scene. Or, it may be possible to use the 

also possible to subsamplc the target scene video data (in 60 control scene to operate a video camera to acquire filmed 

accordance with the control scene content) as it is being scenes from which a target scene can be constructed. In 

acquired to effect control of a displayed fihned scene. particular, this can be accomphshed if the content of the 

Methods for accomplishing such subsampling are described control scene is the same as that of the filmed scene. The 

in, for example, U.S. Pat. Nos. 5,185,667 and 5,444,478, the absence of the target scene is not greatly detrimental, since 

disclosures of which are in corporated by referenc e herein. 65 the user is only seeking to sweep the video camera over all 

j^eTis^ of the content of the panoramic scene, rather than attempting 
to perform precision camera work. 
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However, it need not be the case that the filmed sceaes that displaying the entire panoramic scene as the target scene 

from which a composite target scene is constructed ar^ will cause features in the target scene to be too small to be 

acqiiired pMrJouse^of the yisualuscrinterface. If the visu J easily discerned). In this case, the visual user interface could 

^^tscijntgrfafie is th? be implemented so that when the control scene reaches a 




^gjengreSBb'I^St'S'^Sl^ffiS^ ^ ^^ 5 predefined location near a boundary of the target scene 

(referred to hereafter as "a predefined distance from a 

_ boundary of the target scene/' though it is possible to 

Further, when a visual user interface according to the implement this embodiment of the invention so that part of 
invention is used to control a video camera,Ci5IeiIherrafl^ the control scene may already be extending outside of the 

fim^^entationriOvliicKtHe^c^^ lO boundary of the target scene at this point), the area of view 

on-the3fly3)rIaSri£5pleme^ of the target scene moves as necessary to maintain that 

5:eneriilb6nstructed^l)efore^ predefined distance from the boundary of the target scene, 

the-contemiofltliOomp^itOc^^ The above may be useful to better match the control scene 

Dewiyla^uiredrfilmed^^cenesriSIli^^ target scene sizes and resolutions to the display type 

selectLv^ylto^redi^^ 15 ^scd. 

6^ therewith. 5E6r ex-ample5»only*eve^fQth-filmed-sceTO FIGS. 6 A through 6C illustrate operation of the visual 

^y ^beoised to lupd^te;jthSj£oln'ff5git&^ Or, a record coidd be user interface 200 (FIG. 2) in accordance with an embodi- 

maintained of the time of the most recent update of each ment of the invention in which the area of view of the target 

defined part of the composite scene, and only filmed scenes scene 201 can move as necessary to keep the control scene 

corresponding to "stale" parts of the composite scene (i.e., 20 ^02 at least partly surrounded by the target scene 201. In 

parts of the composite scene that have not been updated for FIGS. 6A through 6C, for aid in illustrating this embodiment 

greater than a predetermined duration of time) used to of the invention, a panoramic scene 601 is shown superim- 

update the composite scene. Additionally or alternatively, posed over the visual user interface 200. As can be seen, the 

"stale" parts of the composite scene can be indicated by a target scene 201 can include only part of the panoramic 

graphical effect which may also be implemented so as to 25 scene 601 at any one time. In FIG. 6A, the target scene 201 

indicate the relative "staleness" of the particular part of the includes content near the center of the panoramic scene 601, 

composite scene (Le., the magnitude of the duration of time and the control scene 202 is positioned within the target 

that the part of the composite scene has not been updated). scene 201 near the bottom of the target scene 202. It is 

For example, the color of a "stale*' part of the composite assumed that the visual user interface 200 is implemented to 

scene can be gradually faded to black and white over time, 3Q allow no portion of the control scene 201 to come closer than 

the boldness of the color of that part of the composite scene a predetermined minimum distance from the boundary of the 

indicating its relative "staleness." target scene 201, and that the distance 602 is greater than this 

In a visual user interface such as the visual user interfaces minimum distance. In FIG. 6B, the control scene 202 has 

200 (FIG. 2) and 500 (FIGS. 5A through 5C), the display of becnmovediip and to the left with respect to the target scene 

the target scene must be continually adjusted as the display 35 201 so that the left boundary of the control scene 202 is a 

of the control scene is moved to different locations within distance 603 from the left boimdary of the target scene 201. 

the target scene. The boundary of the control scene can he The distance 603 is assumed to be equal to the minimum 

changed using conventional GUI methods , as discussed distance; therefore, the content of the target scene 201 is the 

further below. The location of the boundary of the contro l same as in FIG. 6A In FIG. 6C, the control scene has been 

scene within the target scene at any time can b e determine d 40 moved even further up and to the left with respect to the 

trom those methods . Appropnate video data can then be target scene 201. In order to maintain the minimum distance 

selected from the stored video data representing the entire between the boundary of the control scene 202 and the 

composite target scene so that a display of the target scene boundary of the target scene 201, the target scene 201 moves 

is generated in all of the area of the visual user interface that to the left — in coordination with the movement of the 

is allotted for the display of the target scene and that is not 45 control scene 202— with respect to the panoramic scene (in 

being used by the display of the control scene. the visual user interface 200, this would actually appear as 

The control scene in a visual user interface according to the panoramic scene 601 moving to the right in the target 

the invention need not always be confined to the boundaries scene 201). The distance between the left boundaries of the 

of the target scene. However, since the use of a target scene control scene 202 and target scene 201 is still 603. Further, 

to provide context for a control scene is particularly so the distance 604 between the upper boundaries of the control 

advantageous, it may be desirable, where the target scene is scene 202 and target scene 201 is greater than the minimum 

related to the control scene so as to provide such context, to distance (Le., greater than the distance 603); thus, the target 

at least confine movement of the control scene relative to the scene 201 does not move with respect to the panoramic 

target scene so that the control scene is always at least partly scene 601 m that direction. 

surrounded by the target scene. There are two possible 55 Detection of the distance between the boundaries of the 

scenarios that can be addressed. In one, the display of the control scene 202 and target scene 201 can be acoompUshed 

t arget scene is implemented so that the area ot view encom- using conventional GUI techniques. The content of the 

passed by the target scene remains constant . In such an target scene 201 can modified by appropriately stitching 

tiliplementation, limitation ol movement of the conU-ol scene additional filmed scenes to a previously existing target scene 

as described above can be readily accomplished by those 60 201 in the same manner as described elsewhere herein for 

skilled in the art of implementing video displays. In the creating a composite target scene. 

second scenario, the display of the target scene is imple- As indicated above, both the display of a target scene and 

mented so that the area of view encompassed by the target the display of a control scene of a visual user interface 

scene can move. This scenario may be useful in situations according to the invention can be either dynamic or static, 

where it is desired to display a target scene that encompasses 65 For many applications (e.g., videotaping with a video 

only part of a panoramic view to be filmed (perhaps because camera), it is desirable that the display of the control scene 

it is feU that the size of the target scene is sufficiently small be dynamic, since the content of the control scene corre- 
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spends exactly or very nearly to the content of the current 
filmed scene, and it is necessary or desirable to have as 
accurate a depiction of the control scene as possible to 
ensure that the control scene will be controlled to obtain or 
produce the desired sequence of fihned scenes. In applica- 
tions in which the control scene duplicates the current filmed 
scene, the control scene can be generated directly from the 
filmed scene (e.g., from a display feed from a video camera) 
using conventional methods and apparatus. 

In some situations, it may be acceptable to have a static 
control scene display. When the target scene display is also 
static, the content of a static control scene display can easily 
be generated as the content that would otherwise be present 
at that location within the target scene. When the display of 
the control scene is sutic, the control scene can be demar- 
cated in any appropriate manner, such as by, for example, 
displaying an outline of the area of the control scene or by 
maldng the area of the conu-ol scene more or less bright than 
the area of the target scene. The use of a static control scene 
display may be desirable because the generation of a static 
scene can typically be accomplished more easily and inex- 
pensively than the generation of a dynamic scene. WhUe a 
visual user interface including a static control scene display 
can be used for any application, such a visual user interface 
may be advantageously used, in particular, to control a 
lighting device, as described above, since the loss of fidelity 
associated with the static control scene display would typi- 
cally not be detrimental in effecting accurate control of a 
Ughting device. 

The display of the target scene can also be either dynamic 30^ 
or static, the choice usually depending, as described above, 
on the evaluation of a tradeoff between the quality of the 
target scene display and the cost of generating that display. 
A static target scene display can be formed as a composite 
scene as described above, e.g., using a frame stitching 35 
method as described above to create a composite scene from 
multiple filmed scenes. 

A dynamic target scene display can be constructed from 
the sequences of filmed scenes acquired by multiple video 
cameras that each shoot a fixed region of space within a 
panoramic scene at the same lime, the regions adjoining 
and/or overlapping each other. The operation of the multiple 
video cameras is synchronized, as known by those skilled in 
the art, so that frames of video data are acquired at the same 
time (or substantially the same time) by each of the video 
cameras. Such synchronization can be accompUshed in any 
appropriate manner, such as supplying the cameras with a 
genlock reference signal and "free running" the SMPTE 
time code, as known by those skilled in the art. Filmed 
scenes (i.e., frames of video data) acquired at the same time 
from the multiple video cameras can be combined to form a 
dynamic composite scene. Once such a set of temporally 
coincident filmed scenes has been determined, the filmed 
scenes from the set can be combined to form a composite 
target scene by, for example, stitching together adjacent 
and/or overlapping filmed scenes in a manner similar to that 
described above for creating a static composite target scene. 
Frame stitching methods that can be used with the invention 
to create a dynamic composite scene are described in, for 
example, U.S. Pat. Nos. 5,444,478 and 5,187,571, the dis- 
closures of which arc incorporated by reference herein. 
Alternatively, a dynamic composite target scene can be 
created by using the video camera metrics to locate each 
sequence of filmed scenes within the panoramic scene, and 
piecing together the sequences of filmed scenes, based upon 
their locations, so that the sequences of filmed scenes adjoin 
or overlap adjacent sequences of filmed scenes, in a manner 



similar to that described above for creating a static compos- 
ite target scene. 

In some applications of the invention, such as filming of 
movies or television shows, the vistial user interface can be 
used to control a relatively expensive, high quality video 
camera. The use of multiple such cameras to acquire filmed 
scenes for use in constructing a dynamic composite target 
scene would, in many cases, be undesirably expensive. 
Thus, when it is desired that die target scene display be 
dynamic, the cost of producing such a dynamic target scene 
display can be reduced by using cheaper video cameras (e.g., 
single-CCD cameras) to acquire the filmed scenes used to 
construct the composite target scene. An expensive camera 
can be used to acquire the filmed scenes (and, if the interface 
is so configured, the control scene) that are actually to 
comprise the desired video display. For example, the expen- 
sive camera can be mounted on a movable camera mount of 
a tripod and controlled by a position encoder or encoders as 
described above, and the inexpensive cameras can be 
mounted at other locations on the tripod and positioned (i.e., 
aimed in particular directions) so that the filmed scenes 
obtained by the inexpensive cameras can be combined to 
produce the composite scene. Preferably, the nodal points 
about which the inexpensive cameras are positioned are as 
close to coincident as possible. Though the target scene 
display will be of lesser quality than the control scene 
display, this will often not be an important detriment. The 
fact that the target scene display is dynamic rather than static 

is advantageous^in.and-of-itself^— — — ^ 

— ^ j;t|may^alsQihe| i)QssM 
si^e^^merj^ijg^ 
i\^dfe.j^gl^lenSy^^^ 



a ^ ^ljjg^a^^i i^ ^l^^ ^.^ 
distQrtic>ntprrej£.tiqn.t Q^the |fil^ 

example, the above-mentionedU.S. Pat. Nq^5,185,667T 
^Tiny'appFopriate apparatus and methods can be used to 
enable a user to make changes in a control scene. For 
example, •a^^serj:anJbiOnaBlfid~to.rnalce_spatia^ 
^a-controlfscene^(e:^^ 

45"^entation~and/or"locatipn3Qfie^^c^Sol-scene)i;by(2f9 
^xaSglefi^mg aii appropriate-input device fcg:rj^^ro 
keyjboard^trackball^ Jstylus, Joystick^ jjrZTfacikpoinJ™ 
mecfiffism)lto_appropriately'Jchange^I^ 
^controlj^wne-m^acOTTto 
50 fV^onal'graphicd iiser interf ace'(G^U software and han*- 
^-wire^S^jmplementsTiic 
^and-softv^^ used t^^ 

b^e32gp,en^ng;jysteins'sucb as^tlie"JVS^ndowslj!l-operati^g 
(S>^tegQs)5lf, for example, the visual user interface is dis- 
played on a touchscreen, the above-described changing of 
the size, shape, angular orientation and/or location of the 
control scene can be accomplished using a stylus or, 
perhaps, the user's finger. Other apparatus can also be used 
to make spatial changes in a control scene. For example, a 
conventional rocker switch can be used, as can be readily 
imderstood by those skilled in the art, to effect zoom control 
of a control scene, as well as control of the location of the 
control scene. Or, for example, a conventional rotating knob 
can be used, as can also be readily understood by those 
skilled in the art, to effect rotation of a control scene. 
Conventional pushbuttons can also be used to effect desired 
changes in the size, shape, angular orientation and/or loca- 
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tion of the control scene. Many other types of apparatus can on^ghkhithc^isual 

also be used to effect appropriate control of a control scene (o^tlrt constraints 'imposed bv "*tiff viHrn riftvifg 
(including control of visual parameters of the control scene. Changes other than spatial changes (e.g., changes in 
as discussed elsewhere herein), as can readily be appreci- parameters such as focus, tint and/or brightness) can 
ated. For example, a voice recognition system can be used 5 ^^^^ ^ ^ ^^^^^y g^^^ (^l^ich, in turn, effects 
to enable a user to speak commands (e.g, "Zoom'', Pan corresponding control of the video device) using GUI tech- 
Left'', etc.) to effect control of a control scene. ^-^^^g example, the abiUty to control visual parameters 
FIGS. 7 A through 7E illustrate the use of a stylus and g^ch as focus, tint and/or brightness could be implemented 
touchscreen to make changes in the location of a control QUI slider bars that permit continuous adjustment 
scene display that effect corresponding spatial changes in a 10 between two extreme values. Tmt and brightness changes 
fikned scene. In FIG. 7A, the stylus 701 is positioned within ^an also be implemented using conventional mechanical or 
the control scene 702 and dragged across the touchscreen to electronic controls that are usually aheady part of a display 
the right, causing the filmed scene to pan to the right. In FIG. device. A display device for use in displaying a visual user 
7B, the stylus 701 is positioned within the control scene 702 interface according to the invention could also be modified 
and dragged across the touchscreen in a downward direction, 15 to include a mechanical or electronic control for adjusting 
causing the filmed scene to tilt downward. In FIG. 7C, the Qf control scene. A variety of other types of 
stylus 701 is positioned within the control scene 702, physical apparatus, including, for example, rocker switches, 
dragged across the touchscreen to the right, then, transition- pushbuttons and knobs can also be used to effect control of 
ing gradually in a smooth arc, dragged across the touch- visual parameters of a control scene. In general, any appro- 
screen downward, causing the filmed scene to begin panning 20 pnate control mechanism can be used, as can be appreciated 
to the right, then, transitioning gradually in accordance with skilled in the art 

toe arc made by the stylus 701, end by tilling downward In ^ situations, it may be desirable to modify the user 

™- f^^"^ J" positioned withm the control ^^^^^ ^ ^ predetermined manner. For example, when the 

scene 702 and dragged across the touchscreen upw^d and to ^^^^^ j^^^^^ ^ ^^^^j movement 

the left, causmg the filmed scene to pan to the left and tilt 25 ^e more shaky or jerky than desired. Conversely, it may 

upward at the same Ume. In FIG. 7E, tbe stylus 701 is used 'desirable-to create a desired visual ambiance, for 

to draw a circle on the touchscreen outside of the control ^xample-to make a movement of a control scene from one 

scene 702, thus causmg the control scene 702 to move from ^^^J^^ ^^^^^^ ^^^^ 

the location shown m FIG. 7E to a tocauon defined by the ^^^^ ^ accordance with a further aspect of the 

circle (Iks can be miplemented using a bounding box or 30 j^^ention, user movement of the control scene (and. thus, 

or°u^™ % ' '^u ™Pl™'^.n"^f '°,"'%Adobe ^ovementofthe filmed scene) canbe modified according to 

Photoshop™ software) and thereby causing the fllmedscene ^ determined method. Such modification can be accom- 

to move m a similar manner. (The same type of control could ^^^^^ ^^^^ ^^^^ techniques. For example, the 

be effected by touching the sty us at a pomt on the touch- .^^^^ movement of a control scene from one location 

screen outside of the control scene 702.) As can be 3s ^^^^^^^ ^e increased or decreased using the 

appreciated, many other smular types of stylus moves cm be ^ ^^^^ technique developed by Paul Haeberli of Silicon 

used to move Uie control scene 702 and thereby conttol be Inco^-orated, described, for example, at the World 

filmed scene. Further, as can be appreciated, when a display ^.^ ^^^^ www.sgi.com/grafica/dynaAmdex.html 

screen that is not a touchscreen is used to display the visua^ , ^T. 1997) and included as part of tbe SGI 

user interface, such control moves could ako be effected 40 developers Toolbox CD issued in 1997. 

usmg, for example, a mouse (or device havmg similar ^ . , 

ftinclionality, such as a trackbaU) to control the position of I" » ^em, when a movement of a control scene is 

a pointer on the display screen. controlled direcUy by the user (e.g., when the user marks 

„ , » Ml : ; J - ♦u J • *u ♦ 1 Tfti a location or area outside of a control scene to cause the 

Though not illustrated m the drawings, the stylus 701 . ^ ^ t r \ a • 

I J 1 1- J . u »u • el f 1 ^ control scene to move to a new locaUon), a desired control 

could also be used to change the size of the control scene 45 . . ^ j ■ j -.u 

-r/v^ 1. . L J r .L- II scene movement can be produced in accordance with a 

702. (Other apparatus may be used for this purpose as well, j . • j *u ^ ti. * * i 

, ^ \ . . predetermmed method. The creation of a control scene 

such as one or more rocker switches or pushbuttons, as ^ ^ ^ * u • * • n • i A-i^ 

J. J , V 1 -.1 r^TTT *' I, movement from scratch is, typically, smapler than modm- 

discussed above.) For example, convenUonal GUI tech- r * / ul u.u^a 

. , ^ J * f 1^ \i;;„^™oTM cation of a user movement, and can be readily accompUshed 

niques (such as are used m, for example, the Wmdows™ i n j • «t, ^ • • / 

* \ u • 1 * A *u * by those skilled m the art usmg an appropriate mathematical 

operatmg s ystems) could be imple mented that, when the 50 ^ . , ^ u . r % ,u a ■ a u - i ,^ 

t , -n^ / 1 A *u y ' '1^ -^ model of the charactensUcs of the desired physical move- 
stylus 701 feplaced on the outline of the control scene 702J *^ ^ 

then dragged "across the touchscreen, the outline oi' the " 

control scene 702 moves with the stylus, thus changing the The specification of the predetermined method for modi- 
size of the control scene 702. For example, moving the fying or creating a movement of a control scene can be 
stylus 701 toward or away from the content of the control 55 accomplished in any appropriate manner. For example, the 
scene 702 could cause the control scene 702 to become user could select a method from a menu of choices, the menu 
smaUer or larger, respectively, which could, in turn, cause being made available to the user using any of a variety of 

the fihned scene to become smaller or larger, respectively weU-known techniques. ^g ^^ w uL^ 

(i.e., the stylus 701 is used to change the control scene 702 AlSng^g^gJ^S^t^|gg|agEgQ^ 

to effect zooming). The control scene 702 can be made as 6o *^^^^^^S9R|^^^^ffl!m^ 

small as desired, subject only to the limitation that the IjB^^^^^ fl^^m^ l^^He^Sgeras!. Assuming that the 

control scene 702 must remain large enough to enable orientation of each video camera with respect to the scene to 

control by the interface device used to change the control be filmed is known, each camera orientation could be related 

scene 702 and/or to the constraints imposed by a device to the orientation of a target scene of the visual user interface 

(e.g., the lens of a video camera) being controUed.^SimilSly, 65 with respect to the scene to be filmed. Then, as a contr ol 
rtheicontrQl" scene ^702 ^^^t^^^ffid&^I^^l^p^^ desirc!^^ s cpne is changed within the targe t scene, each of the vide o 

aibj eel only^^to tlfe'lim ititiim Mj^^ disp la^y s^^en cameras is controlled as appropnate lo falm ttie rcfflon 
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corresponding to the control scene . Alternatively, changes in made. The display of the time scale can be implemented so 

the operation ot me video cameras can be referenced to one that the time scale appears to scroll through the annotation 

of the video cameras, and that video camera controlled space 904 as time passes during fihning or display. The 

direcUy using the visual user interface. The visual user display of the time scale can also be implemented so that the 
interface could also be implemented to include a display of 5 times in the time scale correspond to lime codes (e.g., the 

the fihned scene bom each of the video cameras, as weU as weU-known SMPTE time codes) a^ociated with particular 

a mechanism to enable specification of the video camera vuleo data (e.g., a particular video frame or frames). Time- 

from which a filmed scene is to be acquired at any given identified annotations (even when the tune associated with 

time. Such multiple video camera control can be particularly annotation only approximately identifies the time to 
useful, for example, in fihning a live event such as a sports lO ^'^•i annoUUon corresponds) are extremely useful to, 

^^^^^ for example, enable someone not lamibar with a videotape 

« . , «^ . ■ ' . ^ to identify the content of the videotape, and, further, the 

HGS. 8A and 8B Uhistrate the visual user mterface 500 ^^^^.^^ of particular content within the videotape. TTie time 

(HGS. 5A through SQ. ■modified to mclude a display of a a\Mstiit>:d in HO. 9 displays the mimber of-^noving 

current filmed scene 801 in addiUon to the displays of the ,^ p,Q ^i^mes, seconds and 

control scene 502 and taigel scene 501. As mdicated above, 15 ^.^^ ^ ^ ^^^^ 

the content of the control scene 502 can be the same as the ^. ^^^^^^^ ^^j^^^ ^ ^ 

content of the cu^ent filmed^ene 801. f desired (and as f^^ ^^^^ .^^^^^^ ^, ^ 

d ustrated m FIGS. 8A and 8B), the display of the curren ^^^^ ^^^^ ^, ^ 

fltoed scene 801 can be unplemented so that the current ^^^^ j o„j„ ^ ^e used. Hough useful, a 

fihned scene 801 cannot be changed(e.g., moved rotated, or 20 necessarily be present in the annotation 

made larger or smaller) by a user (Tlie current filmed scene ^ ^^j^ ^ annotation 

801 Deed not necessanly be implemented m such fashion, ^ ^ r *u . «* 

. „ . . »_ 1 as part 01 the annotation, 

however.) Thus, no matter how a user changes the control L, ^ . i^ i .j- 

scene 502, the current fihned scene 801 pro^^des the user a :maotation space 904 can be miplemented using any 
display of the content that is currendy being fihned or « ''PPrapnate .^PPyatus and methods. For example, the 

displayed which remains constant in size, resoluLon, angular \>^"^ """f^. ^ '^^^^^"i T " }°'^'^^\^ 

/ . ■ 1 ■ , f cnn stylus or the user's finger can be used to wnte notes ID the 

onentaUon and location within the visual user mterface 500. , nn>i o u u -,- , r*u • *- 

This can be appreciated by comparing FIG. 8A to FIG. 8B. "^"^^f °" ^P.^^^ ^""^ ^ embodnnent of the mvenUon 

In FIG. 8B, the control scene 502 has been made smaller and '^I'f'"'*;'' whicb shows a stylus 905 Uiat can be 

u -c J 1- u*i J J * *u 1 „„j nir- QA %o used both to make notes m the annotation space 904 and to 

shifted shghtly down and to the left as compared to FIG. 8A. . . . , l 1.1 

11 - - ^ * ui J- 1 f *u ^ * A\^^A cfti change the control scene 902. The touchscreen enables 

Provision of a stable display of the current filmed scene 801 =, , r.u • 

, /.,r i L*t. i*u *i ascertamment ot the patterns wntten m the annotation space 

can be useful, for example, when the user makes the control _ _ . f • . j j 

F./ . ^ ^ ^. , . . 904. Software embodymg handwritmg recogmtion methods 

scene 502 so small withm the target scene 501 that it is j * • * * *i- #♦ a 

difficult to discern the content of the control scene 502 (and, to mterpret the patter^ as words, numbers 
thus, the filmed scene), when the user has rotated the control 35 and symbols. Touchscrecns and handwriting recogmUon 

cnrt * A 1 «c.»,- ;c- r^^.Ar.^ software that have previously been developed and used m 

scene 502 a great deal, or when the user is movmg or . i j * l r i i j- ** i 

, cA'^ computational devices such as, for example, personal digital 

rotatmg the control scene 502 rapidly. . , , , j* -i f *T 

^ . . assistants (PDAs) can be used to implement an annotation 

FIG. 9 iUustrates a visual user interface 900 m accordance ^p^^^ -^^ ^ interface according to the invention, 

with another aspect of the invention in which a visual user ^ annotation space 904 can also be implemented by 

interface mcludes a space in which a user can make anno- ^ conventional keyboard to type 

tations regarding the device mteraction with the spatial ^^^^ .^^^ amiotation space 904. Other ways of imple- 

region (e.g., when the device is a video device, annotations ^ annotation space 904 are possible, as can be 

regardmg a filmed scene or scenes). For convenience m sedated by those skiUed in the art. However the anno- 

dlustratiDg tlie pnnciples of this aspect of the mvenUon, the ^^^^^ ^ implemented, it is, in general, preferable 

following description will be made with regard to an ^^^^^ interaction with the control scene 902 and the 

embodiment of the mvention m which the visud user ^^^^^^^ 904 via the same input device, so that the 

mterface 900 is adapted for use m controllmg a video device. ^^^^ ^^^^^ ^^^^ ^^^^^ 

Similar to the visual user interface 500 as depicted m presence of an annotation space as part of the same 

FIGS. 8Aand 8B, the visual user mterface 900 mcludes user interface that enables control of a video device enables 

displays of a target scene 901 a control scene 9(^ and a ^ ^^^-jy ^^^^^^^ ^^^^ device and 

current filmed scene 903. Each of the scenes 901, 902 and ^^^^ regarding the content of a sequence of filmed 

903 is displayed and operates as described above. (In scenes acquired or displayed by the video device. A user of 

particular, the scenes 901, 902 and 903 enable a user to interface can quickly move back and forth between 

conuol the operation of the video device.) In addiUon to the controUing the video device via the control scene 902 and 

scenes 901, 902 and 903, the visual user mterface 900 ^^^^ notes in the annotation space 904, thus enabUng 

mcludes an annotation space 904. annotaaons to be made contemporaneously with control of 

The annotation space 904 enables a user to make textual the video device. Moreover, when the device control portion 

notations as the filmed scenes are being filmed or displayed of the interface is implemented in accordance with the 
by the video device. The user can, of course, make any 50 aspect of the invention described above with respect to 

desired textual notations. One advantageous use of the FIGS. 1 through 8B, the ease with which that portion of the 

annotation space 904 is to make textual notations regarding interface enables device control contributes to the ease with 

the content of a filmed scene or scenes as those scenes occur. which the complete interface can be used for contempora- 

As illustrated in FIG. 9, a time scale is automatically neous device control and annotation, because the user is left 
displayed at the leflhand side of the annotation space 904. 65 with more time and attention to devote to making annota- 

The presence of the time scale in the annotation space 904 tions. In contrast, previous ways of annotating a sequence of 

facilitates identification of the time at which annotations are filmed scenes as they are acquired either necessitated two 
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people (one to operate the video camera and the other to take 
notes), or unduly interfered with the operation of the video 
camera. 

Additionally, the use of a touchscreen, stylus and hand- 
writing recognition software in a user interface for enabling 5 
annotations of a sequence of filmed scenes provides wide 
latitude in expression, thus avoiding a limitation of previous 
systems that enable selection of an annotation from a 
predetermined set of "canned" annotations. Such a user 
interface also avoids the problems associated with voice jq 
annotation systems, e.g., dismptioo of the event or scene 
being filmed, and difi&culty in seardiing through the anno- 
tations. (However, a visual user interface according to the 
invention can, if desired, be implemented with a voice 
aimotation system.) j5 

A visual user interface according to the invention can also 
be used to play back a sequence of filmed scenes that has 
previously been acquired using the visual user interface. 
Recorded camera metric information that correlates posi- 
tions of the movable camera mount with filming directions, 20 
orientations and/or positions of the video camera during 
acquisition of filmed scenes can be used during playback to 
locate and scale the filmed scene properly with respect to a 
displayed target scene, so that, for example, the sequence of 
filrned scenes can be viewed in the context of a target scene 25 
that represents a panoramic scene within which the filmed 
scenes were acquired. As can be appreciated, use of the 
visual user interface according to the invention during 
playback provides a much enhanced viewing experience as 
compared to that obtained using conventional camcorders or 30 
television monitors. 

Additionally, a visual user interface according to the 
invention can be used for editing or selective playback of the 
previously acqiured filmed scenes. In general, the content of 
the control scene can be controlled in any manner as 35 
described above to identify particular content from the 
sequence of filmed scenes to either display or store. For 
example, a stylus and touchscreen can be used as described 
above wiUi respect to FIGS, 7 A through 7E to position the 
control scene at one or more locations, an appropriate user 40 
interface mechanism (e.g., any of a variety of appropriate 
graphical user interface mechanisms, such as GUI pushbut- 
tons or pulldown menus) being used to specify that filmed 
scenes having content that is partly or entirely subsumed 
within the area defined by the specified control scenes be 45 
stored or displayed. The stylus and touchscreen can also be 
used to specify, rather than locations, particular types of 
movements (e.g., pans, tilts, zooms) of the control scene, the 
user interface mechanism being used to indicate that 
sequences of filmed scenes corresponding to such move- 50 
ments be stored or displayed. The control scene can also be 
controlled to specify any desired combination of control 
scene locations and/or movements to specify filmed scenes 
to be displayed or stored. 

A visual user interface according to the invention can also 55 
be integrated with a video device that can be automatically 
controlled without user intervention (e.g., a video camera, 
such as the Sony EV1-D30 video camera). An appropriate 
user interface can be used to enable a user to, if desired, 
specify that the video device operate in accordance with a 60 
predetermined method for automatically controlling the 
video device (e.g., motion -tracking or color- tracking 
methods). For example, when a visual user interface accord- 
ing to the invention is used to control the operation of a 
video camera, such integration of automatic video camera 65 
control with the visual user interface can be useful to enable 
a camera operator to specify automatic operation of the 
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camera during a period in which the user desires to make 
extensive annotations regarding the scenes being filmed. 

Various embodiments of the invention have been 
described. The descriptions are intended to be illustrative, 
not limitative. Thus, it wiQ be apparent to one skilled in the 
art that certain modifications may be made to the invention 
as described above without departing fi-om the scope of the 
claims set out below. 

I claim: 

1. A visual user interface for use in controlling the 
interaction of a device with a spatial region, comprising: 

means for displaying a target space and a control space, 
wherein an overlapping portion of the control space is 
within the target space, the content of the overlapping 
portion of the control space being substantially the 
same as the content of the target space that would 
otherwise be present at the location of the overlapping 
portion of the control space, such that the content of the 
target space provides context for at least the overlap- 
ping portion of the control space; and 

means for enabling a user to change the state of the 
display of the control space, wherein the device is 
controlled in response to the state of the display of the 
control space; 

wherein the target space comprises a moving video image 
of a target area defined by the boundaries within which 
the device is capable of interacting with the spatial 
region and the overlapping portion of the control space 
comprises a moving video image of the portion of the 
target space with which the device is interacting at any 
given time; 

wherein the target space comprises an area too large to be 
displayed at one time by the visual user interface and 
wherein the means for displaying is configured to 
display a portion of the target space at any given time 
and to change the portion displayed, if needed, in 
response to movement of the control space relative to 
the target space by the user, so as to di^lay a portion 
of the target space that includes the overlapping portion 
of the control space, even if the control ^ace is moved 
by the user to an area within the target space that lies 
outside the portion of the target space originally dis- 
played by the visual user interface. 

2. A visual user interface as in claim 1, wherein the device 
comprises a video camera, the target space comprises a 
target scene, and the control space comprises a control 
scene; and wherein the control scene comprises a scene 
currently being filmed. 

3. A visual user interface as in claim 1, wherein the means 
for enabling a user to change the state of the control space 
is adapted to enable the user to change the size of the control 
space with respect to the target space. 

4. A visual user interface as in claim 1, wherein the means 
for enabling a user to change the state of the control space 
is adapted to enable the user to change the shape of the 
control space with respect to the target space. 

5. A visual user interface as in claim 1, wherein the means 
for enabling a user to change the state of the control space 
is adapted to enable the user to change the angular orienta- 
tion of the control space with respect to the target space. 

6. A visual user interface as in claim 1, wherein the means 
for enabling a user to change the state of the control space 
is adapted to enable the user to change the location of the 
control space with respect to the target space. 

***** 
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